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Described here is ja research project being undertaken 
to address two points: the relationship of the fidelity of program 
replication to program outco^nes at user sites; and the extent to 
which fidelity can occur in programs with more than a few replicates. 
The findings will have bearing on the usefulness of the Research, 
Develo{Hiient, and Diffusion (RD & D)' Model, which has been criticized 
on the grounds that it does not take into account the modification or 
^reinvention** of programs that occurs when they are replicated at 
other sites. In the current study, eight programs are being 
examined — four in education and four in the criminal justice field. 
Fidelity of replication is to be determined using Hall and Loucks*s 
methods of measuring the numbe^^r and importance of program components 
implemented. Data have already been collected by telephone interview 
with program developers and imj^lementers and will also be gathered 
during site visitations. The relationship between fidelity and 
program outcomes will be analyzed using a multiple regression 
approach. Although the research es at ah early stage of data 
collection/ it can already be concluded that specificity in detailing 
program components is critical in conducting both evaluation and 
implementation research. (Author/JM) 
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To what extent is. the fidelity of a social program's 
implementation related to the outcomes that program produces? 
Can social technologies be disseminated on a large scale such 
that they are implemented with fidelity? These two questions are 
of increasing importance given the recent movement towards de- 
centralization of program initiatives in the present adminis- 
tration. . 

This decentralization is largely based on politicrl motives. 
However, a body of research /exists to support the decentralr 
ization of social programming. This research points out the 
general failure of efforts based on the "Research, Development, 
and Diffusion model" or RDAD (Havelock, 1969) to succeed in 
fostering program implementation. The RD&D model, which guided 
much of the Federal social programming of ^the 1960's, h^s five 
main features (House, Kerins, & Steele, 1972): 

1 ) There is a rational sequence of activities- which moves 
from research to development to packaging. 

2) Planning must occur on a large scale. 

3) A divisio'n of labor separates roles and functions in the 
overall process* 

4) A j^assive cjnsuraer awaits acceptance of the innovation if 
it is delivered properly. 

5) A high initial development cost is necessary to eventual 
success. 

Much recent researc'h has found fault with this model* The 
futility of disseminating complex social innovations as "canned 
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packages" through brochures and other written materials has been 
repeatedly demonstra tod( Fair wea ther , Sanders, & Tornatzky, 1974; 
Fullan & Pomfret, 1977). Treating practitioners as passive con- 
sumers has also been demonstrated to be a waste of time and maney 
(Gross, Giacquinta, S Bernstein, 1 97 1 ) . Practitioner involvement 
in implementation decisions seems to be a virtual prerequisite 
for successful implementation (Fullan & Pomfret, 1977; Tornatzky, 
Fergus, Avellar, Si Fairweath^r, 1981). Exacerbating these prob- 
lems is the fact that programs often undergo some modification or 
"reinvention" by users in order to accomodate programs to 
specific sites (Rice 8l Rogers, 1930). 

However, th'e momentum of the modification or "reiri- 
ventionist** movement may be impelling us to throw the HD&D baby 
out with the bath water, a position cogently stated by Datta 
(l98l). k good portion of Datta*s arguement rests on her 
critique of the large scale RAiND study of federal programs sup- 
porting educational change (Herman & McLaughlin, 1978). The HAND 
research is perhaps the most influential of the re i nv en t io ni s t 
studies. According toDatta, the RAND findings that "mutual 
adaptation" (of site and program) generally characterizes suc- 
cessful implementation is suspect* Although the authors claimed 
that actual program replication rarely occured, their implemen- 
tation outcome measure was "the extent to which projects met 
their own goals" (Berman & McLaughlin, 1977, Vol. VII, p. 50). 
This definition of implementation builds the occurrence of 
adaptation into the results. Secondly, the "programs" examined 



by the RAND study were never specified sufficiently to enable 
"high fidelity" implementation. Finally, the "massive infusion 
of federal dollars", reputed to characterize the Office of 
Education programs which were studied averaged' $100.00 per pupil 
per year across programs* This is hardly a great expense when 
the extent and the complexity of the changes which were to be 
produced is considered* 

In summary, Datta conclu^^^^hat the RD&D model for social 
programming was not dir.ectly tested by the RAND research despite 
claims to the contrary* Similar criticisms can be levelled .at 
other research supporting the modification point of view. 
HjajjjTe_s_,_4i J-nLpiem-en-t^ flawed to subs- 

tantiate research claims (Scheirer, 1981), and progi;am parameters 
are often left poorly specified or entirely unspecified (Bveland, 
Rogers, A Klepper, 1977)» 

It is true, t.hat the milieu of social programs is hardly as 
rational a place as was previously assumed. Au thjors'" who have 
pointed out the need for disseminators and administrators to take 
interpersonal and organizational dynamics into account have made 
valuable contributions to the study of implementation (e.g. Yin, 
1978). However, it may be that a need still exists to develop 
programs through careful and valid research, and to disseminate 
programs which have been demonstrated to attain beneficial out- 
comes for clients with sufficient fidelity to produce these same 
outcomes at the user sites. 

We would argue that the connotations of "research, develop- 



ment, and demonstration" need not include those assumptions 
demonstrated/ to be inadequate. Replacing policies based on these 
assumptions with policies supported by re inv en tionist research 
produces a modified RD&D approach* This approach encourages 
research and development by prac t i tdoners , utilizes active, full- 
time change agents to disseminate rigorously evaluated programs, 
and pays attention to o rgan iza t io nal- env i r o nm en tal dynamics shown 
by past research to be crucial for successful impien\en tat ion • 
This modified HD&D approach, presently utilized by two federal 
agencies, is the focus of the^ present research. 

The intent of this research has been to develop highly- 
specific, behavioral descriptions of eight social programs dis- 
seminated by th ese ag enc ies . These descriptions will then be 
used to address the two issues posed at the beginning of this 
paper: 1) the relationship of fidelity to program outcomes at 
user sites; and 2) the extent to which fidelity can exist in 
programs with more than a few replicates. By- using these 
highly-specified behavioral descriptions, it is hoped that ti.e 
findings of this research will bear more directly on questions 
concerning the usefulness of the RD^SD model than previous studies 
inthisarea. 
Program Fidelity 

The development of the behavioral instruments followed a 
methodology pioneered by Hall., Loucks, and their assoc iat es(Hec}c, 
S tiegelbauer , Hall, & Lou'cks, 1981), working in the area of 
educational innovation. These researche-rs have conceptualized 
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social programs as consisting of a finite number of components or 
parts.. Program fidelity may be defined following this schema in 
-terms, of the number of components actually imjilemented at a site. 
This may , be expressed as' a percentage (number of components 
implemented/numbe-r of components in the program). .Comparisons 
can then be' made both within program and with other programs with 
different numbers of components. 

■ A second major .concept in the Hall and Loucks met-hodology 
involves .the importance of various components. Some components 
are more important(to various actors, for various purposes) , than 
other components. Hall and Loucks have contrasted the core com- 
ponents of a program with its related components. (We. prefer the 
terminology essential vs non- ess'en tial in order to tie the com- 
ponent to an exogenous r e f e ren t s uc h as outcomes). Also, com- 
ponents can be implemented in a variety of ways. Hall and Loucks 
call this phenemenon "component variation". These variations may 
be considered ideal, acceptable, or unacceptable byvarious 
actors (disseminators, ad ftin is t ra to rs , practitioners) depending, 
on their perspective. Variations of components are implemented 
in different identifiable patterns or configurations. 

Hall and his associates have used thi4 framework to study 
the implementation of numerous educational innovations. To our 
knowledge, the research discusued in this paper is the first to 
use this methodology in other areas of social programming outside 
of education. One other study has used this methodology to 
examine fed eral- level dissemination activity (Network, 1981). 



However this study was also concerned solely with education. In 
addition, these iresearchers have chosen to use a different level 
of component specificity than that ut,ilized/in the present' 
research. Our program descriptions range from 56-105 components, 
while those of t^he Network, and Hall and has associates are 
generally in the 5-20 range. 

Research Methodology 

Our research is a three-phased effort. The fi-i-st phase in- 
volved the identification of adopter's through a semi-structured 
phone interview concerning thedr reasons for prog4am adoption. 
The second phase of the research, which is currently in progress, 
involves phone interviews withthe adopters identified in the 
first phase. The second phase is concerned with measuring the 
degree of program implementation and rout inizat ion. The third 
phase will involve site visits to a sample of the organizations 
interviewed during the second phase. 

Innovation Selection " ^ ' 'i 

Programs from two federal agencies, one agency dealiAig with 
education, and one agency Jn the criminal jus ^*ce field, were 
selected. The educational agency was the Na t lonal -Di f fus ion Net- 
work(NDN). The goal of'this agency is the widespread adoption of 
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innovations that are validaled- by the Joint Dis^mii 
Panel. The NDN uses both innovation develope/rs and regional 
facilitators as change agents. The review process acts as a 
quality control on the innovations (Emrick/ Peterson, 4 Agarwala- 



ERLC 



Rogers, 1977). In criminal justice, the Law Enforcement Assist- 
ance Administration Exemplarary Projects Program also requires 
on-site" evidence of effectiveness before a program is designated 
as e*xemplarary and information is disseminated. 'Both agencies^ 
only disseminate information on programs that are known to have 
produ.ced desirable outcomes at the original site*' Variation in 
effectiveness across replicates of a given progra'm can then be 

^ attributed to differences in program implementation at the site^s. 
The first step in se^lecting innovations for study involved 
^tlie reading of materials describing all the NDN prograras(about | 
150 programs^ .and LEAA exemplarary pro j ects( about 35 programs). 
Four " innovat ions from each field were chosen that m^t the follow- 
ing criteria: innovations had to be in the dissemination phase 
for at least two years, innovations had to have a reasonable 
chance of having at least 20 replicates, and innovations had to 
be ''organization-wide^C either involve multiple units of the 
organization and/or demand a good deal of interaction with the 

- organization's environment). Information on* replicates of the 
innovations were obtained from two sources. Developers of the 
innovation were contacted regarding information they had concer- 
ning adopters of their programs. The second source of infor- 

' mation was a three percent random sample select on of the approp- 
riate organization unit(police department, court, prison, or 
school). Adopters "were located through a semi-structured inter- 
view. In this phase of the research, the "organization** was con- 
sidered to be the implementing unit. In the second phase of Our 



required in order to 



research, the "organization" was considered to be the unit in 
which the innovation was housed. The organization can thus dif- 
fer between Phase I and Phase II of our research in the criminal 
justice area. As J • D . Eveland ( 1 981 ) has remarke.d, the choice of 
the appropriate organizational unit is very important. Our^ 
experience substantiates this^point. For this research, con- 
sistency across innovations* in defining the unit of analyses is 

e.nable across - in novation analyses. Jlain- 
taining consistency jtras ther«fo,re the first' priority^in the 
operational definition ^ ''organization". 
Development cf the Implementation Measure 

The second phase of our research involves an addition'al 
interview with each adopter previously identified in Thase I.- 
This interview focused on the implementation of the innovation in 
the adopting organization. Other issues that we are in- 
vestj.gating which will not be^ d iscussed in this paper involve tho 
c^oncept of ijout inizat ion. Of special interest to us is the mag- 
nitude and type of relationship be tween^ impl ementation and 
rout inizat ion. 

In order to measure program implementation it is first 
necessary to specify what the program is and what it involves* 
To do this, we are Using what we hope tb be an ideal (or at least 
an acceptable) variation of the methodology used by Hall ^nd his 
associates. The first step entailed reading all written infpr- 
mation on each of the programs. Two\ staff members then visited 
each innovation developer. These researchers observed either the 



ariginal site of the innovation or the site used for. dis- 
semination purposes. During tfi^se visits, intensive conver- 
sations were held with the developers. These conversations were 



recorded on tapeT 
likely see if he o 



These conversations concerned what one would 
she visited -an innovation adopter.. This type 



of ques_ti-on-, -used Ciy Hall and his associates, focuses the 
developer's attention on t^e^ qbse rvab 1 e behaviors the innovation 



entails. Discus sj orf also included how the typical client. of the* 
innova/tion would interact with various otjher actors involved in 
the innovation. At mbst of these site visits, the programs were 
no longer being run diWectly by the origijnal developers of the 
program, ^^nd conv er Sja t ions with administrators, front-line staff, 
and clients were also recorded. This procedure follows the cbn- 
cern of Hall and his associates that descriptions of programs for 
the purposes of implemeni;kt ion measurement should not* be limited 
to .the developer's perspective. ' 

The second step in our process, involved the separation of 
all the information we had obtained into distinct, identifiable 
components. This process involved one of the staff ^/^jto had gone, 
on the developer/site visit ari^d one additionil researcher. The 



first part of this proce,ss entailed re-reading all the program 
materials. The tapes of\he conversations were heard by both the 
staff who had gone on the vVsitVnd an additional sta'ff member. 
While listening to the ta pe \t rl^d ing the materials, all ident- 
ifiable activities were put on\o cards.' These cards were then 
sorted into categories of innova^iipn-specif ic activities. Dup- 
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lication between activities were eliminated. One of the people 
who had^ visited the developer then took responsibility for the 
I developmen t of the rest of^^the measure.*^ This involved 
'Eliminating further redundancies in the activities and making 

\ - . \ ' ^ . ^< , • r ^ 

Ipure that each component en tai 1 edx only one or a few related 
/^c t ivit ies . \ It is at this point that pur measurement d eve'.opnien t 



s^tarted .to diverge from the strategies used by Hall and his 

/ * 

associates. ]>ue to the limited number of program adopters we 
w.ere working withj it was decided to not interview users of the^ ^ 
program to obtain the variations. Instead, one of the people who* 

drxd the site visit generated the variations. These were review^ed 

• '^^ ^ > ' 

by other researchers on the team. These ^initia^l component lists' 

\^ 

and variations ranged in length from 40-116 components. Extreme 



ca^e was taken to insure a comnrvon level of specificity across v 

\ • \ \ 

innovations. 'Thus, ^variability in the act^^al number of com- 
ponents identified for each progrMt re^presents true variabili^ty^^ 
in. the -Vevel of the complexity of the innovation. 

These component lists were then sent to the . d ev el oper s . , The 
developers were give^two tasks-. The firsf was to respond to a 
list of the components. DevelojJers we'^re asked to label the coin- 



pone'nt as either "essential", "non-essential,", or "not relevant*^ 

\ 

to their program. That is, did the developer view the component 
as critical^ o'r necessary for the d evelop.e.r^to construe an adopter 
as a true replicate of the^initial .innovation? ^Developers were ' 
also asked to decide whether the component^wt^ required in order 
'to obtain the validated outcomes ,for the program./ The second 
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Itask that we asked develope'rs to undertake was the rewriting of 
our variations/ based on their experience and thoughts concerning 
what constituted an ideal, acceptable, or unacceptable variation. 
Data-gathering -involved a phone fj^grview with developers which 
covered every variation of every component. It should be men- 
tioned that in cases where there was more than one develop-er^t a 
site, each developer was asked to assist in these two tasks. 
Hall and his associates have argued cogently for the importance 
of taking multiple perspectives into account in this process. 
Therefore, for one-half of the innovations', program users at the 
original site were asked to perform the same tasks as the 
developers. In revamping j:he instrument to reflect respondent's 
comments, every effort was made to preserve both perspectives. 
In addition, components were "tagged" from both the developer and 
users* view, as to whether the component was essential, non- 
essential, or not relevant, and either important or unimportant 
to outcomes. 

Thus, eight separate fidelity interviews were construe ted 
with their own component and variation lists. Example component 
items from several innovations are included in Table 1. Readers 
should note the similar level of specificity across program 
categories. We are just completing the piloting of this pha'^e of 
the research. 

The interview strategy we have been using is fairly open- 
ended. We ask the user jf the inno\fation about their use of a 
particular category of activity ( a d, p t of components). As they 
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talk, we code the variation they are using under the appropriate 
component. After we have coded every component in a category, /e 
ask the user whether we have missed anything, and whether they^ 
are doling anything "different or unique" with regard to that set 
of components. This is then noted as the amount of modification 
that has taken place in that category. 

Respondencs in this phase are users who have been identified 
a:, nost familliar with the day to day running of the program. -In 
addition, inter-coder reliability is assessed by having t»^o re^ 
searchers code a sample, of interviews. To obtain a measure of 
in terresponden t validity, a second respondent will be interviewed 
at a sample of sites. \ - - 

Site Visits and Collection of Outcome Data 

A 50% sample of the adopters of each of the innovations will 
be site-visited following the telephone interviews. These adop- 
ters will be chosen from the high and low regions of the fidelity 
range, obta ined from these interviews. Fidelity will be cal- 
culated in a number of different ways. First, the percentage ^f 
components im p 1 em en t ed/ the total number of components will be 
computed. This score will enable comparisons across programs. A 
^program-specific calculation of fidelity will also be carried out 
by assigning a score of 2 for components implemented i.deally, 1 
for components implemented acceptably, and 0 for components imp- 
lemented unacoeptably . Both these calculations will be used to 
select high and low fidelity sites to visit. 

Site visits will not be exact replications of phone inter- 
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views, sincd changes in implementation status can occur In the 
interim between the phone interview and the site visit. We will, 
try to minimize the impact of these changes on our measures by 
asking respondents wh]a t has happened on-site since we talked to 
them last. The site visit will then be used to contrast re-sul-'ts — 
of observational methods of data collection with those of phone 
interview methods. 

A second major purpose of the site visits will be th« col- 
lection of outcome data for each program. These will be the 
original validated outcomes of the program. Many of the 
innovations require their adopters to collect this outcome data. 
Data Analysis and Questions to be Answered 

The major questions to be answered concern the relationship 
between fidelity and outcomes, and whe^ther certain configurations 
of components produce higher levels of outcomes than other con- 
figurations. The relationship between fidelity and outcomes will 
be analyzed using a multiple regression approach. Program 
effectiveness (ouccome levels) becomes the criterion variable and 

each of the components become the predictors. The components 

/ 

could be considered in a binary sense (presence oi- absence) or as 
a three-point continuum (ideal=2, ac ceptable= 1 , unacceptable=0 ) . 
The standardized beta weights for each of the components will 
indicate the importance of that particular component to the out- 
come(s). The multipfe R will indicate the magnitude of^the 
relationship between implementation and program effectiveness. 

The innovation configurations cannot be scaled. However, a 
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non-parametric approach can be used. Each configuration can be 
given a different nominal code number and outcome scores can be 
divided into ordinal categories of high, medium, and low 
effectiveness. Chi-square analyses can be used to determine the 
relative frequencies of various conf ipura tions for different out- 
come levels. ^ 

It will be recalled that data were collected from developers 
concerning which components they considered important for 
producing outcomes. The above analysis will suggest which c.qm- 
ponents are empirically important for producing outcomes. These 
two sources will then be compared in order to determine to what 
extent developers are aware of components essential to" produce 
outcomes. If developers are not aware of which components 
produce outcomes, this would suggest major faults in* even a 
modified RD&D model, since disseminators would presumably not 
emphasize "non-essential" components* The abo.ve chi-square anal- 
ysis would speak to the, usefulness of the concept of con- 
figurations. / 
Summary and Conclusions , / 

The wave of recent research and commentary critical of the 
classical RD&D policy model has already influenced program 
policies to a considerable extent. Although many of the points 
raised in the " re inv en tionis t" literature are well taken, the 
essential assumptions of the RD&r model have not been directly 
addressed by research. Our research is an early step in this 
dir ec tion* 
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Unfortunately this research is at an early stage of data 
col 1 ec tion and we cannot yet directly address the issues raised 
here today. However^ we have already learned a great deal from 
the process. Some of what we have learned relates directly to 
J.D, EvelSind * s( 1 981 ) earlier comments concerning the com- 
munalities between evaluation' research and implementation 
research* Specifically, specificity in detailing program com- 
ponents or irxovation configurations is critical. The social 
scientist conducting implementation research in a post hocymanner 
similar to. our project is extremely limited by the detail of the 
initial innovation's evaluation and the degroe-of specificity 
with which the dissemination efforts were, undertaken. The 
relationship between implementation and outcome must be clearly 
detailed at each step along the way. The demands of this area 
are such that they render previous research paradigms somewhat 
impotent in areas of interest to the applied and/or imp- 
lementation researcher. The black box that is a successful 
innovation is no longer acceptable in and of itself. A hybrid 
evaluation model that clearly ties specific innovation components 
and configurations to spec if ic program outcomes is essential for 
the implementation researcher, innovation developer, program 
evaluator, program disseminator, and program adopter. 
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TABLE 1 



Exam-pie Innovation Components 



Assessment of Clients Interests< 



Criminal Justice. 

Skills/lnteres ts/ Assessed . 

!• Res id en t s /ski 1 1 s/ i n t e res ts are assessed prior to 

beginning" training" and job search. 
h • 'Residents ski 1 Is/ in t e r es ts are not a major concern. 

Jobs are considered as they are available. 

Ed^ucation. ^ 

Career Abilities and Interests: Individualized, 
Sy s t em at ic Ass e ssm en t 

I. Student career abilities and interests are 
and systematically assessed, .using explicit 
U. ^S.'^udent career abilities and^ interests are 

individually or systematically. 
U. /Student career abilities anil interests are not assesed 
' , at all. 



ind iv idua 1 ly 
pr oc ed u r e s • 
not assessed 



Regular Monitoring of Clients Progress^ 



Criminal Justice. - > 

Monitoring of Residents* Progress. 

I. Once a month residents are behaviorally rated concerning 

contracted progress. 
A. Every other month residents are behaviorally rated 

concerning contracted progress. 
U. Every third month or less, residents are behaviorally rated 

concerning contractual progress. 
U. Residents are not regularly rated concerning their 

contractual progress. 

Education. 

Academic Learning Plan: Feedback/Progress* Sessions. 

I. Regular (periodic), 1-to-1 feedback and progress meetings 

are held, with students with reference to. the Academic/ 

Learning plan, at least^once a week. 
A. Regular (periodic) 1-to-1 feedback and progress meetings 

arjB held with students at least once a mjnth. 
U • Regular (periodic) 1-to-1 feedback and progress meetings 

with reference to Academic/Learning plan are held with 

students less than once a m'onth. ^ 



